An investigation of soft-button widgets using sound

We outline a novel approach to human-computer interaction, with soft-buttons using a non-structured touch surface and auditory display. Introduction We have developed a novel software defined interaction widget (human-computer interface component) based on a pseudo-haptic user experience using auditory display. The term pseudo-haptic in this context means that the user can move his or her fingers across a non-structured surface containing touch sensors and in certain areas of this surface, defined by software, the user hears friction-like sounds and it was our hypothesis that users would experience this as if the surface has a structure with different friction in different parts of the touch area. The focus of this work is interaction design using auditory display, looking at the auditory dimension as potentially enhancing the user experience. With the emergence of ubiquitous and wearable computers we need to explore alternatives to visual displays, as users sometime need most of their visual attention on the surrounding environment, rather than on visual display of a computer. For example, using a graphical user interface (GUI) on a handheld computer while walking is difficult, or, quite disturbing when standing on scaffolding on a building site, high above ground. On the other hand, there are many kinds of equipment that can easily be operated in such situations, e.g., walkie-talkies, mobile phones, and various forms of electronic instruments (e.g. Geiger counters, metal detectors). Such devices often have a few fixed real physical buttons or controls that can be operated with fingers, in locations that the user have learnt over some period of time, and, often use auditory display to convey read-outs to the user. The location of controls can be found with the fingers, in relation to the casing of the device and sometimes, if the device is attached to the user’s clothing, in relation to the body. Many handheld devices have real physical tactile buttons that can be located by searching the surface of the device with the fingers. Some more novel devices, such as Apple’s iPod also have semi-continuous controllers such as the circular scroll-and-select mechanism, see Figure 1 (right), using s simple click sound when the user moves a finger around the controller area. Many of the existing handheld devices use some form of auditory feedback, most often different kinds of click sounds that, perhaps, enhances the general tactile experience. Le Journées de Design Sonore 2004 2 Figure 1: Mobile phone (left) and FM radio (center) with tactile buttons. Apple iPod with membrane keys and circular scroll-and-select (right). It might be worthwhile to also consider some real-world situations where we use other senses than vision to carry out actions in the world. Some people touch type, based on a long-term experience that has developed into automatic behaviour locating letters on a typewriter or computer keyboard. Some have learnt to operate machinery, such as a cashier’s till or a telephone, without looking at the keys. We all have, at some stage, fumbled in the dark for a light switch or an alarm clock, and quite often been successful in our actions. Most of us can do things like buttoning a button, tying shoelaces or taking a pencil out of our shirt pocket, or, finding a particular kind of coin in our pocket, without using our visual sense modality. Soft-buttons The idea of software defined buttons soft-buttons has emerged from GUIs and direct manipulation, from Xerox STAR onwards (Kay & Goldberg, 1977; Shneiderman, 1982), and are now an important part of most GUI widget libraries. Many GUIs use soft-buttons extensively, on desktop PCs and laptops (Figure 2), as well as handheld computers (personal digital assistants, PDA, see Figure 3). Softbuttons allow the designer, and in some cases also the end-user, to easily modify, add or remove controls of a software application. The way that users can activate functions represented by soft-buttons varies. On desktop computers the most common way is to move a pointing device, such as a mouse, that in turn indirectly moves a visible cursor on screen into the rectangle or polygon surrounding the soft-button and then activating the soft-button by clicking with the pointing device. Visual soft-buttons are often animated to improve feedback to the user, i.e. when the user clicks, the visual button displayed temporarily changes its appearance so it looks like moving inwards, into the display surface. On other kinds of computers, such as PDAs, the user can point directly to a visual soft-button, either with a handheld stylus or a simply with a finger. Figure 2: Some standard Microsoft Windows soft-buttons Le Journées de Design Sonore 2004 3 Figure 3: Palm Pilot with two different Calculator applications using soft-button widgets. User interface widgets such as these make the design of GUIs highly malleable and flexible, as the designer can represent highly complex underlying functionality with simple graphical symbols that, ideally, look like concepts or entities in the user’s task domain. As such, widgets are defined by software rather than hardware. The same physical display surface can be used for different widgets and layouts at different times, supporting the varying and different needs for potential action by the user to carry out different tasks. These features make soft-buttons very attractive as components of a user interface, both for designers and end-users. The problem we address in this study is how to use other forms of display than vision to create similar affordances (Gibson, 1979; Norman, 1988), in this particular case through auditory feedback, mimicking what it would sound like to touch differently structured surfaces. We investigate the use of auditory display to create an interactive pseudo-haptic experience of soft-buttons using a touch tablet, without any difference in physical surface structure (it has a smooth surface) and without visual display, as input device. A somewhat similar pseudo-haptic approach was investigated by Müller-Tomefelde (Müller-Tomefelde, 2003). In one of his demonstrations differences in surface texture through friction-like sounds in a pen-based digitizer application was investigated. In the commercial world, Apple Computer’s Ink application for handwriting input with a digitizer tablet also attempts to enhance the user experience through the use of friction-like sounds as feedback to the user’s pen strokes with a stylus. What is interesting and inspiring in these previous works is that action makes sound. Use Scenario A possible scenario for a device using auditory soft-buttons is a wearable computer used by, for example, telecoms field service engineers (as outlined by Xybernaut (2003)). On reflecting upon this and having tried Xybernaut’s wearable computers, it is possible and potentially an advantage to have a computer on one’s belt with various peripherals on arms and head but the existing Xybernaut implementation is not particularly comfortable or ergonomic for lengthy use (as seen in Figure 4), and requires the user’s visual attention. Although it is obvious that all information cannot be conveyed by sound only, several operations could potentially be easier if the user’s visual modality was freed up and if input devices could be placed on the user’s body so that manipulations requiring fine-motor finger movement (such as manipulating a Le Journées de Design Sonore 2004 4 touch device) can be done with the arm and hand at relative rest (as seen in Figure 5). The current auditory widget emerged from this particular problem, i.e. how to provide a software definable user action potential without using actual tactile buttons or the visual modality. Figure 4: Xybernaut touch display worn on arm Figure 5: Xybernaut touch display worn on belt Experimental study Based on our experiences from informal probes, one haptic and one auditory, we designed an experiment to investigate if users could detect soft-buttons using auditory display. We wanted to find out how feasible it was for users to carry out the task requested using either haptics, i.e. to draw shapes based on real haptic exploration, or, pseudo-haptics using auditory display. The word exploration is important, as shown by Gibson (1961), e.g. if people were presented with static haptic stimuli, they found it almost impossible to determine the shape of an object, while if allowed to actively explore objects they found the task easy and produced accurate results. There are probably many different ways to investigate if and how users detect the presence and layout of user interface components. We can ask users for verbal reports about what they detect; we can give them specific tasks to locate and activate a particular widget among others while recording speed and accuracy of their performance; we can let users explore an interface and then ask them to choose among a number images of interfaces which one they think was the one they explored; or, we can simply ask our participants to make a drawing of what the detect. If a drawing strongly resembles the actual layout, the result can be is considered to be good. We can count each missing feature or severe distortion to get a measure of what was detected. As our main interest is interactive auditory display, we decided to ask users to draw the layout of soft-buttons they detected when exploring a touch device with their fingers and hearing. The resulting drawings were analysed by the experimenters. A drawing that was similar to the actual soft-button layout would be a good result; if a drawing had little resemblance with the actual layout the result would be deemed to be bad. Le Journées de Design Sonore 2004 5 Prototype system In the pilot study of the pseudo-haptic experience we had used a Xybernaut computing platform with a combination of Macromedia Flash (for interaction) and pd (pure-data, for sound modelling/synthesis). While this had shown promising results, we needed to reduce latency and to get more reliable touch data in real-time. The system used for the results reported here were based on a Dell PC, Pentium III, 800 MHz with 256 MB RAM and a SoundBlaster Live! sound card, and a Tactex touch tablet (see Figure 6), that has the advantage over the previously used Xybernaut device in that it does not have a visual display and it can be connected to any computer with a serial port. We wrote the experiment application in C on a Microsoft Windows 2000 platform with DirectSound support. The sound models followed the general approach for cartoonification of sounds, as developed by the Sounding Object project (Rocchesso & Fontana, 2003), with simple noise sources controlled and filtered in real-time, mimicking the acoustic modal resonances and dynamic behaviour of a real system. Figure 6: Tactex touch tablet Sound design considerations While a real change in texture, touched by a human hand, would seldom produce a click sound we decided to add emphasis to the transition between soft-buttons and surrounding areas by adding tick/tack sounds at the boundaries of the button areas. When moving a finger within an active button area, a friction-like sound was generated. Both velocity and pressure on the input device were mapped to resonant frequencies, bandwidth and loudness. Our choice of parameters for this mapping was based on spectral analysis of real friction sounds, e.g. fingers moving across different paper, plastic and textile surfaces. See Table 1 for the mapping between actions and sound. The interactive area of the Tactex device matches the size of a human hand quite well. From design guidelines for real physical tactile buttons, keypads and keyboards, we extracted some key features and limitations regarding dimensions and spatial layout for controls operated by pressing or touching. Many existing keyboards and keypads have buttons spaced 18 to 22 millimetres apart, based on the average size of an adult hand and distance between fingers. 1 www.xybernaut.de 2 www.macromedia.com 3 pd.iem.at 4 MTC Express from Tactex Inc. 5 The only real physical device where we have observed something similar is on small cardboard boxes with adhesive stickers, where the surface of the box is glossy and the sticker is matte and the edges of the sticker are articulated. Le Journées de Design Sonore 2004 6 With three users, we asked them to explore six different haptic buttons and softbuttons layouts (see Table 2) in random order, and to make drawings of the structures they detected. They explored each layout and then made a drawing, before moving on to the next. The sounds were heard in mono, using headphones. Each user was allowed three minutes to make each drawing. Physical setup of experiment The participants had to put their hand in to a cardboard box, hiding their exploring hand from their vision while carrying out the detection tasks to eliminate indirect visual cues to the layout of buttons. See Figure 7 below. Figure 7: Physical setup of experiment. Discussion and Conclusion As can be seen in Table 3, the participants could fairly accurate detect the pseudohaptic buttons, almost as well as the haptic buttons, within the 3 minutes allowed to explore each layout. The main challenge appears to be asymmetrical layouts and too many buttons with too narrow spacing, a challenge that also is apparent in the haptic condition. We have to recognise that this was the first time our participants were faced with this kind of task, for a very limited time, and the circumstances and similar problems may be observed with participants using a computer mouse for the first time, which can be quite challenging, e.g. moving off the desk to reach a particular icon; moving too much while clicking. From our results we believe that soft-buttons using auditory display is a very promising avenue towards wearable and ubiquitous computing without using the visual modality. We also note that some of the problems when working with an auditory interface is latency, as our hearing is about a magnitude faster than our vision, hence current multitasking operating systems do not accommodate a straight forward implementation approach. It also is interesting that a Sound Object approach requires a sound model to be executed in real-time, and in interaction design when there is no action, the model is making silence (Cage, 1961). Le Journées de Design Sonore 2004 7 Layout No. of buttons Characteristics A 12 This layout is aimed to test an even distribution of equally sized touch areas across the surface. Each softbutton is c. 20 x 20 mm with c. 20mm space between buttons. An interesting issue with this layout is how users will detect the edge-boundaries of the softbuttons along the physical edge of the touch area, where the Tactex device has a real haptic affordance. The physical edge is raised by c. 3 mm and made from aluminium (the touch area is covered by plastics). B 8 This layout is similar to A but with fewer soft-buttons. C 6 This layout is symmetrical and has two buttons of different size, 40 x 40 mm. D 6 Similar to C but with diagonal symmetry. E 5 This is a non-symmetrical layout with three different soft-button sizes: (3) 20 x 20, (1) 40 x 40 and (1) 40 x 80. F 18 This layout has the most soft-buttons 20 x 20 mm, spaced only c. 5 mm apart. It is a grid layout but with some grid locations empty. The overall layout is non-symmetrical. Table 2: Button layouts Action Sound Function No touch Touch area outside button Enter button area Tick Move finger on button Friction sound Exit button area Tack Lift finger off button Tock Select/Activate function Table 1: Sounds used for auditory display of user actions in relation to soft-buttons. Le Journées de Design Sonore 2004 8 Table 3: Participant drawings Layout Participant Haptic Pseudo-haptic A 1